DOCUMENT RESUME 



ED 456 155 



TM 033 218 



AUTHOR 

TITLE 

INSTITUTION 



SPONS AGENCY 

PUB DATE 
NOTE 

CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Klein, Davina C. D.; Yarnall, Louise; Glaubke, Christina 
Using Technology To Assess Students' Web Expertise. CSE 
Technical Report 544. 

National Center for Research on Evaluation, Standards, and 
Student Testing, Los Angeles, CA. ; California Univ. , Los 
Angeles. Center for the Study of Evaluation. 

Office of Educational Research and Improvement (ED) , 
Washington, DC. 

2001-06-00 
22p . 

R305B60002 

CRESST/CSE , Graduate School of Education & Information 
Studies, University of California, Los Angeles, CA 
90095-1522. Tel: 310-206-1532. 

Reports - Research (143) 

MF01/PC01 Plus Postage. 

Computer Literacy; Construct Validity; ^Educational 
Technology; Evaluation Methods; *High School Students; High 
Schools; * Knowledge Level; ^Middle School Students; Middle 
Schools; Online Systems; Reliability; Student Evaluation; 
*World Wide Web 
*Authentic Assessment 



ABSTRACT 



This study investigated the use of an online authentic 
assessment tool to evaluate students' fluency with the World Wide Web (WWW). 
Participants included 120 middle and high school students with a strong 
technology background and a familiarity with navigating the World Wide Web. 
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prior Web knowledge, searching expertise, and finding ability. In general, 
these composite indicators make sense theoretically, and the results support 
the construct validity of each. The indicators appear to be reliable, with 
alphas ranging from 0.88 to 0.71. In addition, these measures match the 
theoretical conceptions grounded in the literature. The paper concludes with 
a discussion of implications for future research in assessing and supporting 
students’ Web learning. (Contains 1 table, 4 figures, and 28 references.) 
(Author/ SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



■O 



d 








Center for Researi 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



Kim Hurst 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



J- 



I U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
DUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

Minor changes have been made to 
improve reproduction quality. 

Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



b 







: d^- 



Using Technology to Assess Students' Web Expertise 
CSE Technical Report 544 



Davina C. D. Klein, Louise Yarnall, and Christina Glaubke ; 

CRESST/University of California, Los Angeles 



1 










T . W 

■ 1 > r 1 


/ 

\ 1 


> * / 
l or 


....... Y .. . . 

■ -W; " '• r - 






if AJ 





U 







1 



<0 









r! , UCLA Center for the Study of Evaluation 



» 

In Collaboration With: 

I, 

^ University of Colorado at Boulder • Stanford University • The RAND Corporation 
University of Southern California • Educational Testing Service 
University of Pittsburgh • University of Cambridge 






r 



Ierjci 



BEST COPY AVAILABLE 






Using Technology to Assess Students' Web Expertise 
CSE Technical Report 544 



Davina C. D. Klein, Louise Yarnall, and Christina Glaubke 
CRESST /University of California, Los Angeles 



June 2001 



Center for the Study of Evaluation 
National Center for Research on Evaluation, 
Standards, and Student Testing 
Graduate School of Education & Information Studies 
University of California, Los Angeles 
Los Angeles, CA 90095-1522 
(310) 206-1532 



O 

ERIC 



3 



Project 1.3 Technology in Action Davina C. D. Klein, Project Director, CRESST/UCLA 
Copyright 2001 Regents of the University of California 

The work reported herein was supported under the Educational Research and Development Centers 
Program, PR/Award Number R305B60002, as administered by the Office of Educational Research 
and Improvement, U.S. Department of Education. 

The findings and opinions expressed in this report do not reflect the positions and policies of the 
National Institute on Student Achievement, Curriculum, and Assessment, the Office of Educational 
Research and Improvement, the U.S. Department of Education, or the Department of Defense 
Education Activity. 



USING TECHNOLOGY TO ASSESS STUDENTS' WEB EXPERTISE 

Davina C. D. Klein, Louise Yamall, and Christina Glaubke 
CRESST/University of Califomia> Los Angeles 

Abstract 

This study investigated the use of an online authentic assessment tool to evaluate 
students 7 fluency with the World Wide Web (WWW). Participants included 120 middle 
and high school students with a strong technology background and a familiarity with 
navigating the World Wide Web. Using our online Web Expertise Assessment and 
accompanying WWW background questionnaire, students were able to demonstrate 
their WWW knowledge and expertise. We then identified important individual measures 
and coded these reliably. Results from principal components factor analysis suggested 
four broad indicators of Web expertise: navigational strategies, prior Web knowledge, 
searching expertise, and finding ability. In general, these-eomposite indicators make 
sense theoretically, and our results support the construct validity of each. The indicators 
appear to be reliable— with alphas ranging from .88 to .71. Further, these measures match 
our theoretical conceptions grounded in the literature. We conclude our paper with a 
discussion of implications for future research in assessing and supporting students' Web 
learning. 

With over 1.5 billion Web pages available and the number of pages on the 
World Wide Web (WWW) increasing at a rate of 1.9 million pages a day (Lawrence 
& Giles, 1999; "The Web: Growing by 2 million pages a day," 2000), the amount of 
resources accessible to people is increasing at an phenomenal rate. Surfers of the 
Web can engage in a variety of online activities such as searching for out-of-print 
books, looking up government statistics, and participating in online auctions. In 
1997 there was an estimated one-fifth of Americans (57 million people) with Internet 
access (U.S. Census Bureau, 1999). By 2000, the number of Americans online was 
estimated to be 122 million and projected to be 194 million in 2005 (Carr, 2000). But, 
as popular as the WWW is, and as populated as it is with facts, opportunities, and 
purchasing possibilities, searching the Web is often compared to looking for a 
needle in a haystack. Given the overwhelming amount of information that is 
accessible, effective use of the Web has less to do with learning how to surf the Web 
and more to do with learning how to search. 

Technology coordinators, media specialists, and classroom teachers alike have 
been tackling this issue in schools. With 9 million children using the Internet at 



school (U.S. Census Bureau, 1999), estimates of at least 40-45% of American 
classrooms linked to the Internet (Becker, 1999; Market Data Research, 1998), and 
nearly 90% of teachers reporting that they perceive classroom WWW access as 
valuable or essential for their teaching (Becker, 1999), information literacy curricula 
are beginning to appear and teachers are beginning to assign projects to students 
that include research on the Web (Breivik, 1998; Duffield, 1997; Ercegovac, 1998; 
Roblyer, 1998). In fact, teachers report that the most common classroom use for the 
WWW is research, with Web searching having surpassed skills practice to become 
the third most common use of computers by students at school (Becker, 1999). 
Recent studies on how children and adults use the World Wide Web show this 
environment holds both tremendous promise and thorny challenges for educators. 
As a research environment, the WWW has been compared to a universe {Berners- 
Lee, Cailliau, Groff, & Pollermann, 1992) containing a vast number of information 
sites on countless subjects. But this expansive environment also presents a problem 
for educators because much of this information is not designed for children's use 
(Kafai & Bates, 1997). Research focusing on how children can best use the Web 
environment has underscored the importance of developing their Web-searching 
skills or Web fluency. 

We use the terms Web fluency and Web expertise interchangeably in this report. 
By these terms, we mean students' proficiency with the World Wide Web, generally 
developed through training or experience. What characterizes Web fluency? What 
kind of navigational styles, cognitive characteristics, and search behaviors are 
beneficial when looking for information buried somewhere in this web of 
information? Recent empirical research has focused on describing how adults and 
children use the WWW environment. Novices to the WWW's open-ended 
environment face the challenge of defining a proper starting point and procedures to 
complete the information search (Hill, 1997). Empirical research regarding the 
navigational strategies adults use to maintain orientation in the WWW environment 
shows they do so by using the back button and returning to previously viewed pages 
(Catledge & Pitkow, 1995; Tauscher.& Greenberg, 1997). Hill found that the more 
familiarity adults had with the structure of the WWW environment, the more they 
employed problem-solving strategies, such as integrating new information, taking 
varied viewpoints on the information they found, and extracting the relevant details. 
Hill also found that unfamiliarity with the WWW environment corresponded with 
prolonged attempts to form queries, define search options, and find one's place in 
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the system. Finally, as expected, research on expert searchers suggests that search 
efficiency is also important (see, for instance, Salterio, 1996). 

Although this research was predominately with adults, it also, provides the 
rough outline of children's developing expertise as Web searchers. Like adults, in 
order to prevent disorientation, children use the back button and frequently return to 
"landmark" pages that provide several links to other locations (Fidel et al., 1999). 
Even the youngest children can scroll through Web sites and use hyperlinks to surf 
the Web, and older children can distinguish among different search engines and use 
Boolean operators (Kafai & Bates, 1997). However, research has demonstrated 
that— compared with adults— children have particular difficulty forming effective 
queries and scanning search results (Bilal, 1998; Fidel et al., 1999; Kafai & Bates, 
1997; Schacter, Chung, & Dorr, 1998). These difficulties appear to be related directly 
to children's lower level qf jiteracy, as most problems stem from off-topic queries, 
misspelled queries, najtural language queries, and reluctance to spend time scanning 
search results. Studies of children and adolescents using the WWW show that these 
users have the same difficulties they have in other information database 
environments; examples of these difficulties include constructing effective queries, 
scanning search results pages critically, monitoring the progress of a search, and 
developing broad assessment strategies to determine the relevance of Web sites 
(Bilal, 1998; Borgman, Hirsch, Walter, & Gallagher, 1995; Fidel et al., 1999; Kafai & 
Bates, 1997; Kuhlthau, 1996; Marchionini, 1989). 

While teaching students how to search for information is surfacing as an 
important goal, and recent research has begun characterizing children's searching 
abilities, little has yet to be said about the assessment of these newfound capabilities 
(see Schacter et al., 1998; Schacter, Herl, Chung, Dennis, & O'Neil, 1999, for notable 
exceptions). For guidance in developing an evaluative rubric for WWW fluency, we 
turned to research conducted with other open-ended search systems, such as library 
and hypertext databases. 

Researchers in these areas have described the ideal sequential structure of 
search behavior. Good searching occurs as the searcher learns more about the 
information environment and repeatedly revises a query to adapt to that 
environment. Bates (1989) coined the phrase "berrypicking" to describe how adults 
adjust their search goals and queries as they review new information in a database, 
selecting relevant information as they proceed through a search. Rosenberg (1996) 
described the orientation phase in a hypertext environment as a process of 
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exhausting various paths, a process that ultimately leads the searcher to establishing 
the structure of the information environment. Guthrie and Dreher (1990) outlined 
five phases of a search, including goal formation, category selection, information 
extraction, integration, and recycling. Some have described how a searcher's initial 
understanding of a question will influence navigational patterns. Depending upon 
how open or focused the initial question is, searchers may employ such approaches 
as browsing, focused searching, and random discovery (Carmel, Crawford, & Chen, 
1992; Cove & Walsh, 1988; Marchionini, 1989). 

These descriptions suggest that the best searchers tailor their queries to the 
information environment. Previous research has then tended to measure how 
quickly and precisely searchers accomplish this feat. Expert database searchers find 
information more quickly, use more precise queries (such as Boolean operators), and 
cover more information overall (Salterio, 1996). Thus, the literature suggests two 
different types of outcome measures: information selection and search efficiency. 
Better searchers can be identified by their ability to find more relevant information 
for their query and by less time being devoted to finding good information. 
Interestingly, the literature does not equate good searching with few steps, but with 
many steps, since good searchers tend to review and revisit more information than 
poorer searchers. 

As we investigated how best to assess students' Web fluency, we chose to 
include measures that illustrated the outcomes of students' searches (e.g., precisely 
what they find and how efficiently they find it), and the process involved in 
achieving those goals (e.g., the kinds of navigational techniques they use, their 
searching sophistication), as well as the prior attitudes students possess regarding 
the World Wide Web itself. We thus created a prototype assessment tool to begin to 
explore important factors in students' World Wide Web searches. Our tool assesses 
students' fluency with the World Wide Web by using measures collected during an 
authentic, performance-based assessment task. 

Our research addresses two critical issues. First, we attempt to model the 
creation of a quality assessment instrument to be used in measuring important Web 
skills. Although researchers agree as to the importance of these Web skills, 
measurement instrumentation is sorely lacking. Second, our research approach 
involves operationally defining these technology skills. Using principal components 
analysis, we try to outline various specific factors that make up the construct we call 
Web fluency. We thus define that which we are attempting to measure. 
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In order to investigate the validity and reliability of our assessment tool, we 
conducted a study with middle and high school students. Our research approach 
was to study these experienced Web users to identify a set of student measures that 
would allow us to better understand and evaluate students' facility with the Web. 
Multiple measures— captured both during student searches and as the results of 
these searches — were employed by the use of our online assessment system and its 
associated automated data-logging and scoring capability. These measures were 
then analyzed in order to gain a better understanding of the important constructs 
underlying students' Web expertise. 

Method 



Participants 

One hundred twenty middle and high school students from three schools 
participated in this study. Schools were part of the Department of Defense school 
system and were participants in a large-scale program geared toward innovative 
uses of technology in the classroom. Due in part to this participation, students had 
strong technology background, had access to computers and the Internet in the 
classroom, and were familiar with navigating the World Wide Web. Students' grade 
levels ranged from 7th through 12th grade. The sample was 57% female, with an 
ethnic breakdown as follows: 61% White, 18% African American, 3% Latino, 2% 
Native American, 1% Asian American, 13% of mixed ethnicity, and 3% other. 

Instrumentation 

In order to investigate students' Web expertise, or fluency with the World Wide 
Web, we created the Web Expertise Assessment (WEA). This online assessment 
automatically captures both process and outcome data that are subsequently coded 
and scored (by raters and by computer) in order to characterize students' WWW 
fluency. 

WEA features four important functions: an online search engine, a Web-based 
information space, a navigation toolbar, and an automatic logging capability. In 
appearance, WEA pages resemble the World Wide Web, and the WEA 
interface — including the WEA navigation toolbar at the top of each page — looks like 
Netscape (see Figure 1). The closed, content-controlled WEA information space 
created for this study features approximately 500 pages of information. The toolbar 
includes buttons for navigating back, forward, and home, and buttons to initiate a 




Figure 1. Sample WEA page. 

search, to add a bookmark, to view bookmarks, and to conduct a find-word-in-page 
search. Using this toolbar to navigate, students can use WEA's search engine to 
search through the information space. WEA automatically logs all keystrokes and 
mouse clicks, creating a permanent and extensive database record of all student 
performance. WEA assesses students' expertise with the WWW by presenting 
students with authentic search tasks (such as preparing for a school research report) 
and asking them to find relevant information in a closed Web-based environment. 
Students are then asked to "bookmark" pages they judge relevant to the task. 

We collected two kinds of measures using our online tool. Some measures were 
directly quantifiable from the log data gathered during students' use of the WEA 
system. For example, the number of searches used to find information, some usage 
of navigational techniques to move through the information space (such as the 
number of times the back button was used), the number of bookmarks created 
throughout the search, and the number of steps taken to complete the search were 
automatically extracted from the data log. Other measures were coded by human 
raters following data collection. Examples of these kinds of measures included the 



quality of students' search terms, certain navigational techniques (such as the 
number of redirected searches), the quality of students' individual bookmarks, and 
the overall quality of students' bookmark sets. 

We selected 17 individual measures of performance from WE A, based on prior 
research (Bilal, 1998; Catledge & Pitkow, 1995; Fidel et al., 1999; Hill, 1997; Kafai & 
Bates, 1997; Salterio, 1996; Schacter et al., 1998; Tauscher & Greenberg, 1997). 
Because we custom-developed the software, we could operationalize many of the 
search behaviors identified as useful in distinguishing effective from less effective 
searchers (e.g., use of the back operation). We present data on student performance 
on these various measures in the following section; for clarity, we define some of the 
measures briefly in Table 1. 

WWW attitudes questionnaire. In addition to searching for particular 
information using WEA, students completed a WWW questionnaire addressing 
their attitudes regarding the World Wide Web. On a scale of 1 ( 1 really don t agree), 



Table 1 



Definition of Online Measures 



Measure 


Definition 


Number of steps 


The number of steps a student took to complete the task. Each mouse click 
counted as one step (e.g., selecting a page to visit from a hit list, conducting 
a keyword search, returning to the home page). 


Number of good searches 


Each individual student search was rated as a good search if it was on topic 
and included keywords. Boolean operators, or synonyms. 


Number of redirected 
searches 


Searches were categorized as redirected if a student attempted a new search 
before visiting any page identified by the initial search s hit list. 


Quality of search 


The quality of a student's searches was also judged as a set, with a score of 0 
indicating no keyword searches or completely off-topic searches, a score of 
1 indicating at least oneon-topic keyword search, a score of 2 indicating 
more than half the searches were on-topic keyword searches, and a score of 
3 indicating at least one Boolean keyword search. 


Bookmark score 


Each page a student bookmarked was assigned an individual score, with a 
score of 0 indicating an off-topic bookmark, a score of 1 indicating a 
bookmark on topic but not relevant to the student's task, a score of 2 
indicating a bookmark peripherally relevant to the task, and a score of 3 
indicating an on-topic and directly relevant bookmark. 


Average bookmark score 
Quality of bookmarks 


The mean bookmark score. 

Overall quality of a student's bookmarks with respect to the task prompt. 
This was measured on a scale of 0 (irrelevant response set), 1 (fair response 
set), 2 (good response set), and 3 (excellent response set). 


Search efficiency 


Ratio of the number of good bookmarks made by a student to the total 
number of pages visited. 
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2 (1 don't agree), 3 (I'm neutral), 4 (I really agree), and 5 (/ really agree), students were 
asked to indicate how much they agreed or disagreed with statements such as "The 
information on the World Wide Web is not very useful/ Although students were 
considered to be experienced Web users (due to their teachers' involvement in a 
schoolwide technology program), we hoped this questionnaire would further 
explore individual differences in students' Web attitudes. 

Procedure 

During each testing session, researchers followed a basic test administration 
procedure. First, students filled out the WWW questionnaire, intended to measure 
student attitudes about the World Wide Web. This form was read aloud while 
students completed it. Researchers then introduced the Web Expertise Assessment 
to students. Students were seated individually before computer screens while a 
research team member walked them through a practice search procedure. There was 
one research team member per class. During this initial presentation, students were 
taught to use WEA's navigation toolbar — using each of its buttons at least once 
during the practice search — and were reminded how to conduct searches using 
keywords and Boolean operators. After reviewing WEA's operation, researchers 
gave students their search task assignments. Students were told they had 20 minutes „ 
to bookmark relevant pages and to complete their searches. During the WEA 
administration, researchers reminded students to bookmark relevant pages. At the 
end of the 20-minute period, researchers asked students to conclude their searches 
by returning to the WEA home page. All data were logged online and subsequently 
coded. 

Results 

Preliminary Analyses 

Web expertise assessment. Interrater reliability on items rated by humans 
using detailed scoring rubrics was computed by double-coding 25% of responses; 
reliability was found to be acceptably high on all measures (see Table 2). In addition, 
interrater reliability for coding of bookmark relevance (used to then score bookmark 
measures) was calculated at .97. The complete list of WEA measures, including 
interrater reliabilities (when applicable), means, and standard deviations, is shown 
in Table 2. 
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Table 2, 

Descriptive Statistics for Web Expertise Assessment Measures (N = 120) 



Measure 


Interrater 

reliability 


M 


SD 


Average bookmark score 3 


.97 b 


1.82 


0.82 


Efficiency of search 


.97 b 


0.21 


0.19 


Highest single bookmark score 3 


.97 b 


2.48 


1.00 


Number of good bookmarks made 


.97 b 


1.94 


1.69 


Number of good searches 


.98 


1.97 


2.05 


Number of information pages visited 


— 


11.40 


8.83 


Number of revisited information pages 


; — 


4.04 


8.26 


Number of steps in search 


— 


93.00 


54.70 


Number of times back button used 


— 


17.20 


12.40 


Number of unique keyword searches 


— 


0.69 


0.89 


Number of redirected searches 


.99 — 


2.33 


2.40 


Quality of bookmark set 3 


.97 


1.84 


1.00 


Quality of in-page searches 3 


.75 


0.92 


1.07 


Quality of keyword searching set 3 


.99 


1.69 


0.81 


Total bookmark score 


.97 b 


11.20 


10.40 


Total number of bookmarks made 


— 


5.96 


6.47 


Total number of searches 


— 


4.83 


3.13 



Note. Dash indicates computer coding of data. 
a Coded on a scale of 0 to 3. 

b Reported alpha is for coding of bookmark relevance only; following this 
human rating, calculation of each of these measures was automated. 

As may be seen in Table 2, students performed a mean of about 4.83 searches in 
the 20-minute testing period. Students performed a mean of 1.97 good searches, with 
about a third of these searches (0.69 searches) being keyword searches. In addition, 
students redirected 2.33 other searches— deciding to retry a new search rather than 
visiting any page identified by the search's hitjist. These redirected searches 
demonstrated that students were able to browse a search output list and determine 
whether or not a more refined search was necessary. The quality of students'; 
searches was also judged as a set; students' mean overall quality searching score was 
1.69 (on a 0 to 3 scale, with a score of 2 indicating more than half the searches were 
on-topic keyword searches). These searches led students to visit on average about 11 
unique pages, with about 4 of those pages being revisited over the course of the 





session. Revisiting pages is an orienting technique used by experienced searchers to 
navigate through an information space. Use of the back button is another orienting 
technique used by experts; students used this technique often clicking the back 
button on average 17.2 times during the course of their search. Having arrived at an 
information page, only about one third of students (37%) used the in-page search 
feature appropriately. In sum, students completed an average of 93.0 steps 
(excluding use of back button and revisited pages). 

Because students were told to bookmark as many pages as were relevant and 
useful for answering the prompt, there were many ways to characterize their finding 
ability. Like their searches, the pages students bookmarked were rated "good if 
they were on topic and directly relevant to the search prompt. Of the total number of 
bookmarks made per student (M = 5.96), about one third of these pages were 
considered good by our raters (M = 1.94). Students' average bookmark scores 
ranged from 0 to 3 (a score of 2 signified a bookmark peripherally relevant to the 
search prompt), with the mean bookmark score of 1.82,. However, when we 
examined each student's highest bookmark score, we found the mean to be 2.48. In 
addition, we found that 75% of students made at least one good bookmark. This 
indicates that most students were able to find at least one good page and identify it 
as such. Also, when we calculated overall scores by summing across all bookmark 
scores, the mean student overall bookmark score was 11.2. On a scale of 0 to 3, 
students' mean quality bookmark score (the quality of the bookmarks a student 
made when assessed as a set) was 1.84, with a score of 2 denoting a good, but not 
excellent, response set. Finally, on a scale of 0 to 1, students' mean searching 
efficiency rating was .21 indicating that— of all the pages students chose to 
visit— about one fifth were bookmarked appropriately. 

WWW attitudes questionnaire. Results from the WWW attitudes 
questionnaire, including specific items, means, and standard deviations, are shown 
in Table 3. In general, students' responses were as might be expected from 
experienced Internet users. Only 6% of students agreed or strongly agreed that the 
information on the WWW is not very useful, whereas 61% of students reported 
disagreeing or strongly disagreeing with the statement ' There is not a lot of detailed 
and in-depth information on the World Wide Web." Given the erratic nature of 
information on the WWW, it is not surprising that half the students chose a neutral 
response, neither agreeing nor disagreeing with the statement The information out 
on the World Wide Web is accurate or correct." An additional 20% of students 
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Table 3 

Descriptive Statistics for WWW Background Questionnaire Items (N = 120) 



Item 


M 


SD 


The information out on the World Wide Web is accurate or correct. 


3.09 


0.82 


The information on the World Wide Web is not very useful. 


2.10 


0.87 


There is not a lot of detailed and in-depth information on the World Wide Web. 


2.28 


0.93 


The World Wide Web is helpful in finding information. 


4.16 


0.82 



Note. Scale: 1 = / really don't agree to 5 = / really agree. 



disagreed or strongly disagreed with this statement. We take this as an indication 
that these students were critical consumers of Web information. Regarding the 
helpfulness of the WWW, students were positive in their beliefs, with 83% of 
students agreeing or strongly agreeing that the WWW is helpful in finding 
information. 

Principal Components Analyses 

We selected from these various WEA and WWW questionnaire measures a set 
of 13 non-overlapping items whose statistical distributions were appropriate for 
further analyses (see Table 4). Scores for negatively worded questionnaire items 
(items 2 and 5 on the WWW attitudes questionnaire) were reversed, and all 
variables were standardized. A principal components factor analysis using varimax 
rotation was then used to examine further the relationships between the various 
variables. Table 4 shows the four significant factors that emerged from the principal 
components analysis (confirmed by Scree plot examination), accounting for a total of 
71.2% of the variance. Three variables loaded highly on Factor 1: Use of the back 
button, the number of steps in the search (the number of backs and number of 
revisits were not included in this calculation), and the number of revisited 
information pages were all included in this Navigational Strategies factor. Factor 2, 
students' Finding Ability, included three high-loading variables: students' average 
bookmark score, the quality of their bookmark set, and the efficiency with which 
they searched. All four WWW attitudes questions loaded highly on Factor 3, the 
Web Attitudes factor. Finally, Factor 4, termed students' Searching Expertise, 
included high loadings for variables measuring the number of good searches, the 
quality of the search term set, and the number of redirected searches. 
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Table 4 

Rotated Factor Matrix for Web Expertise Assessment Items (N = 120) 



Item 


Factor 1 


Factor loadings 
Factor 2 Factor 3 


Factor 4 


Number of times back button used 


.97 


-.02 


.03 


-.02 


Number of steps in search 


.92 


.07 


.11 


.09 


Number of revisited information pages 


.76 


-.07 


.02 


-.02 


Average bookmark score 


-.04 


.92 


.05 


.04 


Quality of bookmark set 


.26 


.87 


.07 


-.06 


Efficiency of search 


-.26 


.85 


.005 


.07 


The World Wide Web is helpful in finding information. 


.09 


-.05 


.81 


.10 


The information on the World Wide Web is not very useful. 3 


.02 


.10 


.80 


.25 


The information out on the World Wide Web is accurate 
or correct. 


-.08 


.02 


.73 


-.22 


There is not a lot of detailed and in-depth information on 
the World Wide Web. 3 


.13 


.11 


.69 


.02 


Number of good searches 


.05 


.05 


-.03 


.91 


Quality of keyword searching set 


-.01 


.01 


-.01 


.77 


Number of redirected searches 


-.002 


-.03 


.13 


.66 


Eigenvalues 


2.54 


2.37 


2.34 


2.00 


Percent of variance explained 


19.5% 


18.2% 


18.0% 


15.4% 



Note. Factor loadings with absolute values of .30 and higher are shown in boldface type, 
a Scores for negatively worded questionnaire items were reversed. 

Based on the results of our factor analysis, four scales were created. Scale 
alphas were high, ranging from .71 to .88. Each scale, its associated items, and its 
reliability coefficient are shown in Table 5. 

Discussion 

Our goal for this research was to explore the constructs underlying students' 
Web fluency — their ability to search for and find relevant information as they 
navigated through a large information Web space — using as participants students 
experienced with the World Wide Web. Using our online Web Expertise Assessment 
and accompanying WWW attitudes questionnaire, students were able to 
demonstrate their WWW attitudes and their searching, navigating, and finding 
expertise. We then identified important individual measures and coded these 
reliably. Results from our factor analysis suggest four broad indicators of Web 



Table 5 

Scale Descriptions, Related Items, and Alpha Reliability Coefficients for Web Expertise Assessment 



Scale description 


WEA items 


Scale 

alpha 


Navigational 

strategies 


Number of times back button used 

Number of steps in search 

Total number of revisited information pages 


.88 


Finding ability 


Average bookmark score 
Quality of bookmark set 
Efficiency of search 


.86 


Background Web 
knowledge. 


The World Wide Web is helpful in finding information. 

The information on the World Wide Web is not very useful, (reversed)! 
The information out on the World Wide Web is accurate or correct. 
There is not a lot of detailed and in-depth information on the World 
Wide Web. (reversed) 


.76 


Searching expertise 


Number of good searches 
Quality of keyword searching set 
Number of redirected searches 


.71 



expertise: navigational strategies, prior Web attitudes, searching expertise, and 
finding ability. In general, these composite indicators make sense theoretically, and 
our results support the construct validity of each. The indicators appear to be 
reliable— with alphas ranging from .88 to .71— with the searching composite 
measure clearly the least stable. In addition, these measures match our theoretical 
conceptions grounded in the literature. 

Results indicate that students experienced with the WWW tended to agree that 
information found on the WWW was useful and detailed. Further, although they 
reported the Web as being helpful in finding in-depth information, they were also 
aware of the presence of inaccurate information. Effective Web users thus begin 
their tasks with strong familiarity with the WWW environment. 

Turning next to searching expertise, students conducted searches, using quality 
keyword sets. Boolean operators, and synonyms. However, as demonstrated in 
previous research, students had difficulty searching: On average, less than half of 
students' searches were rated "good" and — although most students conducted at 
least one on-topic keyword search — the majority of their searches when reviewed as 
a set were not. We believe this difficulty in searching may be part of the reason for 
the search indicator's relatively lower reliability coefficient (a = .71). Students were 



able to redirect their searches, critically reviewing their search output prior to 
continuing their searches. Good searchers, then, use on-topic keywords and Boolean 
operators in their queries as needed, scan their search results, and frequently adjust 
their queries after reviewing incoming information. 

Regarding navigational strategies, students used two techniques showing 
expertise: Students used the back button often and revisited pages to orient 
themselves in the information space. The number of steps in a student's search also 
helped define his or her navigational expertise. 

Finally, although students' bookmark scores were not as high as we might have 
expected and performance on th e finding measures were the least impressive of the 
set, WEA results showed that experienced students could identify (via 
bookmarking) relevant pages, create quality bookmark sets to answer their search 
prompts, and search with reasonable efficiency. We believe that students may have 
been bookmarking excessively (and thus including less than ideal bookmarks) due 
to our continued urges to do so (as pilot testing had shown the need to remind 
students to bookmark frequently). Regardless, the data certainly suggest that Web 
fluency includes the ability to find and identify necessary information efficiently. 

Clearly, some limitations to this study exist. The participants in this study were 
considered experienced Web users (due to computer and Internet access available at 
school); however, formal data on students' Web experience were unavailable. 
Further, more data using additional search tasks would be helpful. Even so, given 
these limitations, our research findings suggest four indicators of Web fluency as a 
basis for future research. We hope future studies will focus on these four areas. 

Conclusions 

In this chapter, we present a set of indices to measure and characterize 
students' Web expertise. We suggest that an online authentic assessment in which 
students are asked to search for information in a content-controlled Web space can 
indeed capture information toward this end. Our assessment system incorporates a 
variety of measures to reliably evaluate students' WWW background experience, 
searching expertise, navigational strategies, and finding ability. We hope to 
contribute to the literature that seeks to identify the characteristics of Web expertise 
among children and to provide educators with some guidelines on how to support 
students' Web learning. Future work will focus on addressing two questions related 
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to the validity of our assessment: (a) Can we distinguish between expert and novice 
Web users using WEA? and (b) Is WEA performance sensitive to instruction? 

Should we begin to answer these questions positively, we envision the use of 
this automated online assessment to evaluate students' Web fluency. In its first few 
years of existence, the World Wide Web has become one of the most frequently used 
computer technologies in school, with teachers accepting the Internet as an 
incredibly useful classroom tool. Assessing students' expertise in this area is 
important because the Internet is now pervasive; at a minimum, classroom 
computers equipped with connectivity to the "Information Superhighway" are 
being used for this purpose. Thus, it is incumbent upon us as educational 
researchers to set out guidelines for teaching how to use the Web effectively, as well 
as to examine the effects of Internet usage on our students. We believe that if we 
give teachers authentic, performance-based ways of assessing these skills early on, 
they will also use these techniques to help teach their students how to become better 
searchers, navigators, and finders within this latest information space. 
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